Surgical instrument detection system and computer program

ABSTRACT

A surgical instrument detection system is provided that can determine the kinds of surgical instruments without special processing, such as application of an optically readable symbol, to the surgical instruments. A surgical instrument detection system 100 includes: an image input section 31 to input an image taken by a camera 1; an object extraction section 32 to clip an object image of a small steel article from the input image; a determination section 33 to input the object image to a learned classification model 331 and determine a kind of the small steel article based on features included in the object image; and an output image generation section 34 to generate an image representing the result of determination by the determination section and output such image to a monitor 2.

TECHNICAL FIELD

The present invention relates to a system to detect a surgicalinstrument used in surgery and the like.

BACKGROUND ART

Many surgical instruments, such as forceps and tweezers, are used insurgery. Since leaving a surgical instrument in the patient's bodyduring surgery is medical malpractice that should not occur, it isthoroughly ensured that the kind and number of surgical instruments arecounted before surgery and whether all surgical instruments are completeis checked after surgery.

However, surgical instruments come in a wide variety of kinds and manyof them look very alike only with a slight difference in shape and size.It is thus difficult even for an experienced nurse to accuratelydistinguish the kinds of surgical instruments by visual observation.

A surgical appliance preparation system is conventionally proposed thatcan identify the kinds of surgical appliances (including small steelarticles) using optically readable symbols, such as bar codes, given tothe surgical appliances (PTL 1). It should be noted that the surgicalappliance preparation system disclosed in PTL 1 is used for preparationof surgical appliances used in surgery before surgery and is not tocheck whether the surgical appliances are complete after surgery.

CITATION LIST Patent Literature

PTL 1: JP 6-142105A

SUMMARY OF INVENTION Technical Problem

However, giving such an optically readable symbol to each surgicalinstrument requires special processing, causing an increase inproduction costs of the surgical instruments. It also takes time andeffort for reading the symbol from each surgical instrument with areader.

It is an object of the present invention to provide a surgicalinstrument detection system that can readily identify the kinds andnumber of diverse surgical instruments without special processing, suchas application of an optically readable symbol, to the surgicalinstruments.

Solution to Problem

To solve the above problems, a surgical instrument detection systemaccording to the present invention includes: an image input section toinput an image taken by a camera; an object extraction section to clipan object image of a surgical instrument from the input image; adetermination section to input the object image to a learnedclassification model and determine a kind of the surgical instrumentbased on features included in the object image; and an output imagegeneration section to generate an image representing the result ofdetermination by the determination section and output such image to amonitor.

Advantageous Effects of Invention

According to the above configuration, it is possible to provide asurgical instrument detection system that can readily identify the kindsand number of diverse surgical instruments without special processing,such as application of an optically readable symbol, to the surgicalinstruments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an overview of a hardware configurationof a surgical instrument detection system according to a firstembodiment.

FIG. 2 is a block diagram schematically illustrating a functionalconfiguration of the surgical instrument detection system according tothe first embodiment.

FIG. 3 is a flow chart illustrating a process flow by the surgicalinstrument detection system according to the first embodiment.

FIG. 4 is a schematic diagram illustrating an example of output to amonitor.

FIG. 5 is a schematic diagram illustrating another example of output tothe monitor.

FIG. 6 is a schematic diagram illustrating a still another example ofoutput to the monitor.

FIG. 7 is a block diagram schematically illustrating a functionalconfiguration of a surgical instrument detection system according to asecond embodiment.

FIG. 8 is a schematic diagram illustrating an example of output to themonitor.

FIG. 9 is a schematic diagram illustrating another example of output tothe monitor.

FIG. 10 is a block diagram schematically illustrating a functionalconfiguration of a surgical instrument detection system according to athird embodiment.

FIG. 11 is a block diagram schematically illustrating a functionalconfiguration of a surgical instrument detection system according to afourth embodiment.

DESCRIPTION OF EMBODIMENTS

A surgical instrument detection system in a first configuration of thepresent invention includes:

an image input section to input an image taken by a camera;

an object extraction section to clip an object image of a surgicalinstrument from the input image;

a determination section to input the object image to a learnedclassification model and determine a kind of the surgical instrumentbased on features included in the object image; and

an output image generation section to generate an image representing theresult of determination by the determination section and output suchimage to a monitor.

According to the first configuration, it is possible to clip the objectimage of the surgical instrument from the image taken by the camera anddetermine the kind of the surgical instrument using the learned model.This allows automatic determination of the kind of the surgicalinstrument without applying special processing to the surgicalinstrument.

A second configuration includes the first configuration, in which theoutput image generation section outputs an image representing the kindof the surgical instrument overlaid on the object image in the takenimage.

According to the second configuration, it is possible to readilyrecognize what sort of surgical instrument each object image is on themonitor.

A third configuration includes the first or second configuration, inwhich the object extraction section clips the object image based oncomparison of a color space vector for each pixel in a background imageand the taken image.

Although the background image has to be obtained in advance, the thirdconfiguration can clip the object image without depending on thebackground.

A fourth configuration includes any one of the first and secondconfigurations, in which the object extraction section inputs the takenimage to a learned object extraction model and clips the object imagebased on features included in the taken image.

In the fourth configuration, object extraction is learned in a pluralityof lighting conditions in advance, thereby allowing automatic clippingof the object image not affected by the lighting conditions, such asshades and color temperatures.

A fifth configuration includes any one of the first and secondconfigurations, in which the object extraction section converts pixeldata of the taken image to data in an HSV color space and clips theobject image based on edge information on hue and saturation.

According to the fifth configuration, it is possible to clip the objectimage not affected by the lighting conditions because hue and saturationare less likely to be affected by shades in comparison with brightnessand an environmental change affecting the whole, such as a change incolor temperature, does not affect the edges.

A sixth configuration includes any one of the first to fifthconfigurations, in which the system further includes a comparisonsection to compare the kinds and number of surgical instruments aftersurgery with the kinds and number of surgical instruments beforesurgery, wherein

the output image generation section generates an image representing theresult of comparison by the comparison section and outputs such image tothe monitor.

According to the sixth configuration, it is possible to compare thekinds and number of surgical instruments after surgery with those beforesurgery, allowing effective prevention of medical accidents of leaving asurgical instrument in the patient's body during surgery.

A computer program as an aspect of the present invention causing aprocessor of a computer to execute surgical instrument detection,wherein the surgical instrument detection includes instructions for:

inputting an image taken by a camera;

clipping an object image of a surgical instrument from the input image;

determining, by inputting the object image to a learned classificationmodel, a kind of the surgical instrument based on features included inthe object image; and

generating an image representing the result of determination andoutputting such image to a monitor.

A surgical instrument detection method as an aspect of the presentinvention executed by a processor of a computer, the method including:

inputting an image taken by a camera;

clipping an object image of a surgical instrument from the input image;

determining, by inputting the object image to a learned classificationmodel, a kind of the surgical instrument based on features included inthe object image; and

generating an image representing the result of determination andoutputting such image to a monitor.

EMBODIMENTS

A detailed description is given below to embodiments of the presentinvention with reference to the drawings. An identical reference sign isgiven to identical or equivalent parts in the drawings to omitrepetitive descriptions. To facilitate the understanding of thedescription, the drawings referred below may be illustrated in asimplified or schematic configuration or may have omitted components.The scale ratio of the components illustrated in each drawing does nothave to reflect the actual scale ratio.

First Embodiment

The first embodiment is described below.

FIG. 1 illustrates an overview of the hardware configuration of asurgical instrument detection system according to the first embodiment.A surgical instrument detection system 100 according to the presentembodiment is a small steel article detection system to particularlydetect small steel articles among surgical instruments and includes, asillustrated in FIG. 1, a camera 1, a monitor 2, a computer 3, a keyboard4, and a mouse 5.

The camera 1, the monitor 2, the keyboard 4, and the mouse 5 areconnected to the computer 3. Although FIG. 1 exemplifies a configurationprovided with the keyboard 4 and the mouse 5, it is possible to omit thekeyboard 4 and the mouse 5 by using a monitor allowing touch screeninput. Alternatively, a computer may be used that includes a microphoneinstead of the keyboard 4 and the mouse 5 to accept directions from auser by voice input.

The camera 1, the monitor 2, and the computer 3 are mounted on a cart 6.The cart 6 is provided with legs 61 having wheels. The cart 6 has a pole62 with an arm 63 mounted on a tip end, and the arm 63 holds the camera1. The arm 63 desirably has two or more rotation axes to freely adjustthe field of view of the camera 1. The cart 6 further includes a table64 on which the keyboard 4 and the mouse 5 can be placed. The cart 6further includes a computer support bench 65 on which the computer 3 canbe installed.

The surgical instrument detection system 100 having the aboveconfiguration can be moved by pushing the cart 6 and is brought into anoperating room and the like to photograph, as illustrated in FIG. 1, aninstrument placement table 7 on which surgical instruments are placedwith the camera 1. The image taken by the camera 1 is sent to thecomputer 3 for detection of the small steel articles by imageprocessing.

FIG. 2 is a block diagram illustrating the functional configuration ofthe surgical instrument detection system 100. It should be noted thatthe block diagram illustrated in FIG. 2 conceptually illustrates thefunctions of the surgical instrument detection system by classifyingthem into blocks and that each block does not have to be actuallyimplemented as individual hardware. It is possible to achieve each blockby executing a predetermined program by a CPU of the computer 3.

As illustrated in FIG. 2, the surgical instrument detection system 100includes an image input section 31 to input an image taken by the camera1, an object extraction section 32 to clip an image of an object that ishighly probably a small steel article from the input image, adetermination section 33 to determine the kind of the small steelarticle from the object image clipped by the object extraction section32, an output image generation section 34 to generate a display imageindicating the result of determination by the determination section 33,and a memory section 36 to store data used for the process by the abovesections.

A procedure by each of the above section is described with reference tothe flow chart in FIG. 3.

In the present embodiment, the object extraction section 32 performspixel value comparison process to clip an image of an object that ishighly probably a small steel article. Accordingly, in the presentembodiment, an image of the instrument placement table 7 is taken withthe camera 1 before placing surgical instruments to be used as abackground image for reference. A sterilized sheet is generally spreadon the instrument placement table 7 to arrange surgical instrumentsthereon. The background image taken before arranging the surgicalinstruments thus includes an image of the sterilized sheet only. Theimage input section 31 stores the background image taken with the camera1 in a background image memory section 361 in the memory section 36(S1).

Then, when instruments used in surgery are set on the instrumentplacement table 7, an image of the instrument placement table 7 is againtaken with the camera 1. The image thus taken is temporarily stored in ataken image memory section 362 in the memory section 36 by the imageinput section 31 (S2). As illustrated in FIG. 4, the taken image isdisplayed on the monitor 2 by the output image generation section 34. Inthe example of FIG. 4, an image is taken where eight pairs of forceps 91are arranged on the instrument placement table 7.

The object extraction section 32 includes a pixel value comparisonsection 321. The pixel value comparison section 321 reads the backgroundimage and the taken image described earlier from the memory section 36to compare an RGB vector of pixels included in each image (S3). Theobject extraction section 32 further includes an object detectionsection 322. The object detection section 322 determines an areacomposed of pixels having an angle made by the RGB vectors of both thebackground image and the taken image or a difference in vectormagnitudes of them greater than a predetermined threshold as an objectof a small steel article (S4). The taken image memory section 362 givesa flag indicating the object to the pixels constituting the object amongthe pixels included in the taken image.

As illustrated in FIG. 5, the result of object extraction may bedisplayed on the monitor 2. In the example illustrated in FIG. 5, anobject border 92 is displayed around each of the eight pairs of forceps91 detected as the objects from the taken image.

When the process by the object extraction section 32 is completed, thedetermination section 33 determines which small steel article theextracted object corresponds to using a learned classification model 331to which deep learning is applied (S5).

The learned classification model 331 is generated by learning manyimages of obtained by photographing various small steel articles. Thesmall steel articles may be classified based on features such as theentire or partial shape and color. A change in the direction of lightincident to such a small steel article greatly changes the brightnessdue to shades and mirror reflection, often affecting the recognitionaccuracy. The images for learning are thus preferred to be taken byvariously changing the conditions of light incident to the small steelarticle.

The determination section 33 inputs each object image clipped by theobject extraction section 32 to the learned classification model. Thelearned classification model decides which small steel article theobject image corresponds to based on the features included in the inputobject image and outputs the result of decision together withprobability of the decision. The probability of the decision is a valueindicating the likelihood that the object image corresponds to the smallsteel article.

When the probability of the decision to the individual object image isgreater than a predetermined value (e.g., 90%), the determinationsection 33 outputs data indicating the kind of small steel articleoutput from the learned classification model as the result ofdetermination for the object image to the output image generationsection 34. Meanwhile, when the probability of the decision is thepredetermined value or less, the determination section 33 outputs“unknown” as the result of determination for the object image to theoutput image generation section 34.

The output image generation section 34 generates an image to be outputto the monitor 2 based on the result of determination received from thedetermination section 33 (S6). As illustrated in FIG. 6 for example, theoutput image generation section 34 generates an image containing a linesurrounding each object and a text (character string) representing theresult of determination and displays it on the monitor 2 overlaid on thetaken image. At this point, if there is an object output as unknown orfalsely recognized, a user manually corrects the result of determinationand fix it. The result is fed back to the learned classification model,thereby allowing update of the model and improvement in recognitionaccuracy.

For example, Pean forceps can be classified by the shape at the tip end(straight/curved) and thus they are displayed with a text, such as“straight Pean forceps” or “curved Pean forceps”. In the example of FIG.6, all object images are determined as “straight Pean forceps”.

As just described, the surgical instrument detection system 100 in thepresent embodiment clips the object images of the small steel articlesfrom the image taken with the camera 1 and uses the learnedclassification model to each object image for automatic determination ofthe kinds of small steel articles. This allows accurate determination ofthe kinds of small steel articles without having to apply specialprocessing, such as optically readable symbols, to the small steelarticles.

Second Embodiment

A surgical instrument detection system 200 according to the secondembodiment has, in addition to the surgical instrument detection systemaccording to the first embodiment, a function of checking whether allsmall steel articles are complete after surgery.

As illustrated in FIG. 7, the surgical instrument detection system 200thus further includes a comparison section 35. In the presentembodiment, after surgery, collected small steel articles are placed onthe instrument placement table 7 to be photographed with the camera 1and, similar to the first embodiment, the kinds and number of the smallsteel articles included in the taken image are detected. The comparisonsection 35 compares the kinds and number of small steel articles beforeand after surgery to detect the excess or deficiency.

In the surgical instrument detection system 200, the memory section 36further includes a prescribed number memory section 363 to memorize thekinds and number of small steel articles before surgery. That is, theprescribed number memory section 363 memorizes the kinds of small steelarticles prepared for use in surgery and the number for each kind. Thesmall steel articles to be used are arranged on the instrument placementtable 7 before surgery to be photographed with the camera 1 and thekinds and number of the small steel articles determined by thedetermination section 33 are preferably memorized in the prescribednumber memory section 363. Alternatively, without performing automaticdetection by the system, the kinds and number of small steel articles tobe used may be input using the keyboard 4 and the mouse 5.

FIGS. 8 and 9 illustrate examples of a screen displayed on the monitor 2in the surgical instrument detection system 200. FIG. 8 is an example ofa screen displayed on the monitor 2 after small steel articles arrangedon the instrument placement table 7 are photographed by the camera 1after surgery. The screen exemplified in FIG. 8 displays a taken imagedisplay area 81, an instrument kind display field 82, a prescribednumber display field 83, a recognized number display field 84, anot-yet-recognized number display field 85, an excess or deficiencynumber display field 86, a scroll bar 87, and a control button 88.

In the instrument kind display field 82, the kinds of small steelarticles are displayed. In the prescribed number display field 83, thenumber of small steel articles before surgery is displayed for each kindbased on the data memorized in the prescribed number memory section 363.In the recognized number display field 84, the not-yet-recognized numberdisplay field 85, and the excess or deficiency number display field 86,nothing is displayed at this point. On the control button 88, a commandof “recognition” is displayed. When an operator clicks the controlbutton 88 after the image taken by the camera 1 is displayed in thetaken image display area 81, the process by the object extractionsection 32 and the determination section 33 are started.

Then, as illustrated in FIG. 9, in the taken image display area 81, thekind of small steel article is displayed by text overlaid on each objectof the small steel article detected from the taken image. At this point,a command of “registration” is displayed on the control button 88. Inthe not-yet-recognized number display field 85, the number of objects isposted that is determined to highly probably correspond to the kind bythe determination section 33. If there is an object falsely recognizedor decided as unknown, a user can correct the result of determination.Then, when the control button 88 is clicked, the result of determinationfor the not-yet-recognized objects is fixed and posted in the recognizednumber display field 84. In the excess or deficiency number displayfield 86, a value is displayed that is obtained by subtracting thenumber displayed in the recognized number display field 84 from thenumber displayed in the prescribed number display field 83. In theexample of FIG. 9, the result of determination by the determinationsection 33 is displayed for the surgical instruments named as “straightPean forceps (S)”, “straight Pean forceps (L)”, “curved Pean forceps(L)”, and “non-toothed tweezers (M)”. When recognition and registrationof all small steel articles to be posted are completed with no problems,“O” is displayed in the excess or deficiency number display field 86.

As has been described, the present embodiment allows automaticcomparison whether the kinds and number of small steel articles coincidebefore and after surgery by image recognition. It is thus possible toprovide a system to automatically check if any small steel article isleft in the patient's body without applying special processing to thesmall steel articles.

Third Embodiment

A surgical instrument detection system 300 according to the thirdembodiment includes, as illustrated in FIG. 10, an object extractionsection 32A instead of the object extraction section 32 in the firstembodiment. The object extraction section 32A extracts an object imagethat is highly probably a small steel article using a learned objectextraction model 321A to which deep learning is applied. It is possibleto obtain such a learned object extraction model by learning many imagesof small steel articles placed on a sterilized sheet on the instrumentplacement table 7.

That is, the learned object extraction model is, different from thelearned classification model used to decide the kinds of small steelarticles, a learning model to determine whether a predetermined pixel(or a pixel area) is a part of an image of a small steel article or thebackground (sterilized sheet).

As just described, the present embodiment is different from the firstembodiment in that the learned model is used for clipping of an objectimage as well.

In the present embodiment, photographing of the background image only isnot required different from the first embodiment. In addition, use of asufficiently learned model as the learned object extraction model allowsaccurate clipping of an object image.

The object extraction section 32A may be provided instead of the objectextraction section 32 in the second embodiment.

Fourth Embodiment

A surgical instrument detection system 400 according to the presentembodiment includes, as illustrated in FIG. 11, an object extractionsection 32B instead of the object extraction section 32 in the firstembodiment. The object extraction section 32B is provided with aconversion section 321B and an edge detection section 322B.

The object extraction section 32B detects an object taking advantage oflow saturation in the HSV color space, such as silver and black, ofsmall steel articles in contrast of sterilized sheets used as abackground that are generally blue. The conversion section 321B convertsthe RGB data of the input taken image to data in the HSV color space.The edge detection section 322B detects an edge (border) of an objectbased on hue and saturation data. The edge detection technique is notparticularly limited because various such technique is conventionallyknown, and for example, the canny method may be used. The edgeextraction based on the hue and saturation data after converting the RGBdata to the data in the HSV color space allows suppression of theinfluence by shades due to a change in ambient light. This is because,although the brightness data in RGB and the lightness data in HSV areaffected by shades, the hue and saturation are less likely to beaffected by shades compared with them.

The object extraction section 32B may be provided instead of the objectextraction section 32 in the second embodiment.

While some embodiments of the present invention are described above,embodiments of the present invention are not limited only to thespecific examples above and may be variously modified. It is alsopossible to embody the present invention by appropriately combining thefunctions described in the respective embodiments above.

For example, although the surgical instrument detection system toparticularly detect small steel articles among surgical instruments isdescribed in the above embodiments, the present invention is alsoapplicable to detection of, other than the small steel articles, lightmetal surgical instruments with a surface finish equivalent to the smallsteel articles or surgical instruments made of other materials such ascarbon or resins.

In the above description, the embodiments of the surgical instrumentdetection system are described as a system implemented in a server or acomputer. It should be noted that embodiments of the present inventioninclude a computer program and a storage medium storing the same tocause a general-purpose server or a computer to achieve the functions ofeach block described above.

That is, all or part of the process by each functional block in theabove embodiments may be achieved by a program. Then, all or part of theprocess by each functional block in the above embodiments is performedby a central processing unit (CPU) in the computer. As an option, it isalso possible to use a coprocessor, such as a graphics processing unit(GPU), a tensor processing unit (TPU), and a field programmable gatearray (FPGA). The program to perform each process is stored in a memorydevice, such as a hard disk and a ROM, and read by the ROM or a RAM forexecution.

Each process in the above embodiments may be achieved by hardware or bysoftware (including the cases where it is achieved together with anoperating system (OS), middleware, or a predetermined library). It maybe achieved by process in combination of software and hardware.

The order to execute the processing method in the above embodiments isnot limited to the description of the above embodiments and the order ofexecution may be changed without departing from the spirit of theinvention.

The scope of the present invention includes a computer program causing acomputer to execute the method described earlier and a computer readablestorage medium having the program stored therein. Examples of thecomputer readable storage medium include flexible disks, hard disks,CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, Blu-ray discs (BD), andsemiconductor memories.

The computer program is not limited to those stored in the above storagemedium and may be transmitted via electrical communication lines,wireless or wired communication lines, networks including the internet,and the like.

REFERENCE SIGNS LIST

1: Camera, 2: Monitor, 3: Computer, 4: Keyboard, 5: Mouse, 6: Cart, 7:Instrument Placement Table, 31: Image Input Section, 32: ObjectExtraction Section, 33: Determination Section, 34: Output ImageGeneration Section, 35: Comparison Section, 36: Memory Section, 100,200, 300, 400: Surgical Instrument Detection System

1. A surgical instrument detection system, comprising: an image inputsection to input an image taken by a camera; an object extractionsection to clip an object image of a surgical instrument from the inputimage; a determination section to input the object image to a learnedclassification model and determine a kind of the surgical instrumentbased on features included in the object image; and an output imagegeneration section to generate an image representing the result ofdetermination by the determination section and output such image to amonitor.
 2. The surgical instrument detection system of claim 1, whereinthe output image generation section outputs an image representing thekind of the surgical instrument overlaid on the object image in thetaken image.
 3. The surgical instrument detection system of claim 1,wherein the object extraction section clips the object image based oncomparison of a color space vector for each pixel in a background imageand the taken image.
 4. The surgical instrument detection system ofclaim 1, wherein the object extraction section inputs the taken image toa learned object extraction model and clips the object image based onfeatures included in the taken image.
 5. The surgical instrumentdetection system of claim 1, wherein the object extraction sectionconverts pixel data of the taken image to data in an HSV color space andclips the object image based on edge information on hue and saturation.6. The surgical instrument detection system of claim 1, furthercomprising a comparison section to compare the kinds and number ofsurgical instruments after surgery with the kinds and number of surgicalinstruments before surgery, wherein the output image generation sectiongenerates an image representing the result of comparison by thecomparison section and outputs such image to the monitor.
 7. A computerreadable storage medium comprising a program causing a processor of acomputer to execute surgical instrument detection, wherein the surgicalinstrument detection comprises instructions for inputting an image takenby a camera; clipping an object image of a surgical instrument from theinput image; determining, by inputting the object image to a learnedclassification model, a kind of the surgical instrument based onfeatures included in the object image; and generating an imagerepresenting the result of determination and outputting such image to amonitor.
 8. A surgical instrument detection method executed by aprocessor of a computer, the method comprising: inputting an image takenby a camera; clipping an object image of a surgical instrument from theinput image; determining, by inputting the object image to a learnedclassification model, a kind of the surgical instrument based onfeatures included in the object image; and generating an imagerepresenting the result of determination and outputting such image to amonitor.